Skip to content

feat: separate external KV outcome and retirement - #2

Merged
feichai0017 merged 6 commits into
mainfrom
feat/external-kv-receipts
Nov 15, 2026
Merged

feichai0017 merged 6 commits into
mainfrom
feat/external-kv-receipts

Conversation

@feichai0017

@feichai0017 feichai0017 commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

An external cache operation may fail while its device or transport still accesses borrowed pages. CacheTransfer separates that logical outcome from a successful physical-retirement receipt. The existing Engine storage job retains its page loans and exact exported views until both are observed.

This is the first implementation step of #1, stacked on docs/hierarchical-kv-cache. It adds no provider adapter or second coordinator.

Owner contract

  • CacheStorage.load and store return CacheTransfer(outcome, retirement). Lookup remains advisory. Standalone callers use transfer.wait(timeout=...); one timeout budget covers both receipts.
  • The Python bridge retains the exact exported tensor views and runner before provider submission. An exception, cancelled/invalid retirement receipt or old bool-future return is conservatively unquiesced: keep backing and page charges, fence new storage work, and retain the job through final Engine drop.
  • A logical failure is reusable only after a successful retirement receipt. No uncertain transfer is replayed. A source retirement receipt is distinct from publishing a complete cache object.
  • MemoryCacheStorage implements the two receipts. Its failed fence stays sticky, and close() reports it without retrying native cleanup.
  • The breaking provider change is recorded in docs/releases.md; the provider guide and external-KV design describe the migration and remaining work.

The correspondence follows vLLM's scheduler/worker separation and delayed block release, and SGLang's registered pools and named state components. Exact source revisions are pinned in docs/external-kv-design.md. Neither reference becomes a dependency.

Imported baseline repairs

These are separate commits from the receipt implementation:

  • Declare CPU-test import dependencies and model source paths inherited by subprocess tests. Use eight equivalent NonZero divisor expressions for current Clippy.
  • Preserve Copy-enum extraction, PyO3 guard errors and inspection hints explicitly on Rust 1.100. No lint is disabled.
  • Join the distributed-HTTP fixture's shutdown watcher before destroying process groups. An early-return control forces the old watcher/group teardown overlap without sleeps; no production HTTP behavior changes.
  • Use nonuniform signed dyadic data for two differently shaped GEMM layout/dtype comparisons. Their exact equality, row, dtype and source-preservation assertions remain unchanged; model arithmetic is untouched.
  • Explicitly coalesce the instantaneous HTTP score fixture's output so it fits the bounded queue without assuming its reader is scheduled between fake model steps. Every token/text/score comparison stays; dedicated overflow and slow-client cancellation controls are unchanged.

Validation

All 5/5 checks pass on head 9f3665b1fce003c9fd793d85584d856b17dc2199:

Gate Result
Linux CPU foundation 313 Rust tests; 2,362 Python passes / 564 explicit skips; installed-wheel CPU example and package/source controls pass
macOS CPU foundation 310 Rust tests; 2,360 Python passes / 566 explicit skips; installed-wheel CPU example and package/source controls pass
Same two CPU jobs, repetition step 18 controls × 50 repetitions on each OS, all pass; 15 storage cases plus three fixture controls
Apple Silicon tests Rust tests and 59 MLX tensor controls pass; no full-model qualification
Checks Current-stable Clippy; Python lint, formatting and typing pass

Commands: uv sync --locked --no-default-groups --group test, uv run --no-sync python scripts/check.py, and the 50-iteration pytest step in .github/workflows/ci.yml. The full local development check also passed, including installed packages. Changed documentation links/anchors and git diff --check pass.

The new controls drive real Engine jobs, page charges and real CPU byte views. They force logical completion before retirement, cancellation, close and final-reference drop with events. A counterfactual that bypasses the retirement gate fails the page-charge assertion; restoring the old daemon watcher fails the joined-watcher assertion. Logs stay outside Git. GPU, cached-model and platform skips are reported, not counted as hardware evidence.

Handoff

  • Commit: 9f3665b1fce003c9fd793d85584d856b17dc2199; base: docs: design hierarchical external KV cache #1 at 10ef0ee. Implemented for review, not independently accepted.
  • Requirement: external-KV P0, Python storage outcome/retirement and exact backing retention. No wire change.
  • Diff against the design branch: 26 files, +715 / −116 lines, net +599.
  • P0 remains partial: native/device execution backing audit, representation identity binding and borrowed-byte budgets remain open. Existing storage-operation/page limits remain in force; no new byte-limit claim is made.
  • OrbitKV, Mooncake Store and LMCache adapters remain unimplemented. No installed GPU gate or qualification is claimed by this CPU ownership sub-step.
  • No OrbitKV/OrbiTE changes, package publication, model downloads or hardware jobs. Stop for review before extending the contract.

@feichai0017
feichai0017 marked this pull request as ready for review November 15, 2026 08:29
Copilot AI balanced review requested due to automatic review settings November 15, 2026 08:29

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@feichai0017
feichai0017 changed the base branch from docs/hierarchical-kv-cache to main November 15, 2026 10:18
@feichai0017
feichai0017 merged commit 8bb189d into main Nov 15, 2026
5 checks passed
@feichai0017
feichai0017 deleted the feat/external-kv-receipts branch November 15, 2026 11:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants